1. Technical Field
The present disclosure relates generally to the process of building software on computer systems and more specifically to techniques for managing build dependencies among software modules to produce reproducible builds.
2. Background Information
In a large scale software development, it is often desirable to create reusable software modules that solve general purpose problems, and then combine these software modules in various forms, to create different deliverable software products. Individual developers, or teams of developers, may each write source code for different software modules. Such source code may be written in high-level programming languages (such as C or Java™), in scripting or dynamic languages (such as Perl or Python), and/or in low-level languages (such as assembly code). Typically the source code is stored in one or more source files held and versioned in a source code management (SCM) system. A SCM system may provide helpful management of source code, for example, managing who is allowed to make changes to the source code, providing the ability to document and annotate changes, providing the ability to roll-back to previous versions if changes are not successful, as well as providing other useful functionality.
A developer may periodically “build” the software module(s) he or she is working on from the source files maintained in the SCM, for example, at completion of coding or at an interim point for testing purposes. Building software typically involves processing of source files containing source code, and possibly other files such as previously built files, libraries and/or data files, using one or more software development tools, e.g., a build tool, a compiler, and/or a linker, to create deliverable software that may be used by others. During building, at least some portions of the source code may be converted into executable code.
A variety of challenges are faced when attempting to build software, even when only an individual software module is being built. For example, even when building an individual software module, a developer must deal with “build dependencies”. Typically, certain portions of software should be built in a certain order, such that some portions of the software should be already present (e.g., already built) before other portions may be built.
Some build dependencies are quite simple. Other build dependencies are far more complex, and may involve interdependencies created by references in a first source file to individual classes, methods, functions, and/or data found in other source files. These complex build dependencies may be expressed in a build-order dependency graph, having nodes and directed acyclic links that represent dependencies between source code modules.
A build tool may be employed by developers to manage the building of an individual software module. A build-order dependency graph is typically provided to a build tool as an implicit representation, for example, in one or more Makefiles or solution files that define a file (or set of files) or a rule name as depending on a set of files, in build commands that specify a build ordering (e.g., “BeforeBuild” and “AfterBuild” in Microsoft Visual Studio™ development environments “solution” files), and/or in batch files that specify a build order.
While these solutions may be sufficient when individual developers are attempting to build individual software modules, they suffer a number of shortcomings when applied to a large scale software development, wherein many reusable software modules are often combined in various forms to create different deliverable software products. Individual software developers often attempt to control the building process for the individual software modules they create, as if those modules had no external dependencies. That is, individual software developers often tend to concentrate solely on the sub-graph of the build-order dependency graph related to their particular software modules. This often creates a hard-to-manage implementation, where Makefiles, solution files, build commands, and/or batch files for individual software modules attempt to control the entire building process for the overall software product. This approach may lead to sometimes hard to locate inconsistencies and errors.
Even if an individual developer tries to properly consider external dependencies, with conventional tools, it may be quite difficult. As discussed above, such dependencies may be expressed implicitly, rather than explicitly, in Makefiles, solution files, build commands, and/or batch files. This obfuscation makes the handling of external dependencies quite difficult.
Further, enforcing standards for interaction between individual software modules during a build in a large scale software development is often problematic, since individual software modules may be built using different software development tools, versions of software development tools, etc.
Finally, conventional tools often fail to allow incorporation in a current build of previously-built “last known good” builds of individual software modules that have not changed. As a result, resources may be wasted rebuilding software modules that have not changed.
Accordingly, there is a need for improved techniques for orchestrating building of software modules to create deliverable software products, including techniques for managing build dependencies among software modules that may be created by different software developers.